Identifying video content via fingerprint matching

ABSTRACT

Methods and systems to identify video content based on video fingerprint matching are described. In some example embodiments, the methods and systems generate a query fingerprint of a frame of video content captured at a client device, query a database of reference fingerprints, determine the query fingerprint of the frame of captured video content matches a reference fingerprint, and identify the video content based on the match of fingerprints.

TECHNICAL FIELD

The subject matter disclosed herein generally relates to the processingof data. Specifically, the present disclosure addresses systems andmethods to identify video content via fingerprint matching.

BACKGROUND

Typically, people watch video content, such as television shows,advertisements, movies, video clips, and so on, via devices that receivea transmission from a content source. For example, a broadcaster (e.g.,HBO® or CNN®), a web server (e.g., YouTube®), a peer-to-peer source(e.g., another device), and so on, streams or otherwise transmits videocontent to various devices capable of presenting the video content, suchas televisions and associated set-top boxes, computing and/or mobiledevices and associated media players or browsers, and so on.

BRIEF DESCRIPTION OF THE DRAWINGS

Some embodiments are illustrated by way of example and not limitation inthe figures of the accompanying drawings.

FIG. 1 is a network diagram illustrating a network environment suitablefor identifying video content via fingerprint matching, according tosome example embodiments.

FIG. 2 is a block diagram illustrating components of a videoidentification system and a query fingerprint generator, according tosome example embodiments.

FIGS. 3A-3B are display diagrams illustrating patches applied to a frameof video content.

FIG. 4 is a flow diagram illustrating an example method for generating aquery fingerprint, according to some example embodiments.

FIG. 5 is a flow diagram illustrating an example method for identifyingvideo content, according to some example embodiments.

FIG. 6 is a block diagram illustrating components of a machine,according to some example embodiments, able to read instructions from amachine-readable medium and perform any one or more of the methodologiesdiscussed herein.

DETAILED DESCRIPTION Overview

Example methods and systems for identifying video content viafingerprint matching are described. In some example embodiments, themethods and systems generate a query fingerprint of a frame of videocontent captured at a client device. For example, the methods andsystems may select multiple patches of the video content and calculate avalue for each of the selected multiple patches using an integral imagetechnique or other similar technique. The systems and methods may querya database of known reference fingerprints using the generated queryfingerprint, determine the query fingerprint of the frame of videocontent matches a known reference fingerprint, and identify the videocontent based on the match of fingerprints.

In some example embodiments, in response to the identification, thesystems and methods may return an identifier for supplemental contentassociated with the identified video content to the client device. Theclient device may use a received identifier to access metadata,event-based content, and/or other supplemental information or contentthat is associated with video content and/or at a time or location(e.g., a frame or block of frames) within the video content, that iscurrently being presented via the client device.

In the following description, for purposes of explanation, numerousspecific details are set forth to provide a thorough understanding ofexample embodiments. It will be evident to one skilled in the art,however, that the present subject matter may be practiced without thesespecific details.

Example Network Environment

FIG. 1 is a network diagram illustrating a network environment 100suitable for identifying video content via fingerprint matching,according to some example embodiments.

The network environment 100 may include a watching station 110 thatreceives video and other multimedia content from a content source 105,such as a broadcaster, web server, and so on. For example, the contentsource 105 may be a broadcaster, such as television station ortelevision network, which streams or transmits media over a televisionchannel to the watching station 110, and/or a web service, such as awebsite, that streams or transmits media over a network 130 to thewatching station, among other things. The watching station 110 includesa reference fingerprint generator 115 that generates referencefingerprints of video content received from the content source 105.

One or more client devices 120 may also receive the video and othermultimedia content from the content source 105, such as via a broadcastchannel and/or over the network 130. The client devices 120 may includetelevisions, set-top boxes, laptops and other personal computers,tablets and other mobile devices, gaming devices, and other devicescapable of receiving and presenting a stream of video and/or othermultimedia content.

In some example embodiments, the client device 120 may include a tunerconfigured to receive a stream of video content and play the stream ofvideo content by processing the stream and outputting information (e.g.,digital or analog) usable by a display of the client device 120 topresent the video content to a user associated with the client device120. The client device 120 may also include a display or other userinterface configured to display the processed stream of video content.The display may be a flat-panel screen, a plasma screen, a lightemitting diode (LED) screen, a cathode ray tube (CRT), a liquid crystaldisplay (LCD), a projector, and so on.

The network 130 may be any network that enables communication betweendevices, such as a wired network, a wireless network (e.g., a mobilenetwork), and so on. The network 130 may include one or more portionsthat constitute a private network (e.g., a cable television network or asatellite television network), a public network (e.g., over-the-airbroadcast channels or the Internet), and so on.

In some example embodiments, a video identification system 150communicates with the watching station 110 and the client device 120over the network 130. The video identification system 150 may receive aquery fingerprint generated by the query fingerprint generator 125 ofthe client device 120, such as a fingerprint of a frame or block offrames within the video content, and query an index of known referencefingerprints generated by the reference fingerprint generator 115 of thewatching station 110, in order to identify the video content by matchingthe query fingerprint with one or more reference fingerprints.

Upon identifying the video content, the video identification system 150may return an identifier for supplemental content (e.g., metadata,event-based information, and so on), such as content contained in asupplemental content database 160, associated with the video content tothe client device 120. Using the identifier, the client device 120 mayaccess the supplemental content from the database 160 and present thesupplemental content along with playing video content, among otherthings. For example the client device 120 may access and presentsupplemental content from the database 160, such as listing or guideinformation for a broadcast channel, metadata associated with playingvideo content, information associated with playing video content, and soon.

Any of the machines, databases, or devices shown in FIG. 1 may beimplemented in a general-purpose computer modified (e.g., configured orprogrammed) by software to be a special-purpose computer to perform thefunctions described herein for that machine. For example, a computersystem able to implement any one or more of the methodologies describedherein is discussed below with respect to FIG. 6. As used herein, a“database” is a data storage resource and may store data structured as atext file, a table, a spreadsheet, a relational database, a triplestore, or any suitable combination thereof. Moreover, any two or more ofthe machines illustrated in FIG. 1 may be combined into a singlemachine, and the functions described herein for any single machine maybe subdivided among multiple machines.

Furthermore, any of the modules, systems, and/or generators may belocated at any of the machines, databases, or devices shown in theFIG. 1. For example, the video identification system 150 may include thequery fingerprint generator 125, frames of video content from the clientdevice 120, and generate the query fingerprints using the included queryfingerprint generator 125, among other configurations.

Examples of Identifying Video Content

As described herein, in some example embodiments, the systems andmethods described herein utilize fingerprints of video content toidentify the video content. FIG. 2 is a block diagram illustratingcomponents of a video identification system 150 and a query fingerprintgenerator 125, according to some example embodiments.

The query fingerprint generator 125 of the client device 120 (or, thereference fingerprint generator 115 of the watching station 110)includes a patch selection module 210 and a value calculation module220, all configured to communicate with each other (e.g., via a bus,shared memory, or a switch). The video identification system 150includes an index module 230, a fingerprint match module 240, and anidentification module 250, all configured to communicate with each other(e.g., via a bus, shared memory, or a switch).

One or more of the modules described herein may be implemented usinghardware (e.g., a processor of a machine) or a combination of hardwareand software. Moreover, any two or more of these modules may be combinedinto a single module, and the functions described herein for a singlemodule may be subdivided among multiple modules.

In some example embodiments, the query fingerprint generator 125 isconfigured and/or programmed to generate a query fingerprint of one ormore frames of video content captured at the client device 120. Forexample, the query fingerprint generator 125 may calculate values ofpatches, such as Haar-like features, regions, portions, and/or otheraspects of one or more frames within the video content. For example, apatch may be a portion of a frame having various different geometries, aHaar-like feature, and so on. In some example embodiments, some or allcaptured patches may each have a different scale and be at a differentlocation within a frame, among other things.

The query fingerprint generator 125 (and the reference fingerprintgenerator 115) generates and/or creates fingerprints for identifyingvideo content from frames within the content. Typically, video contentreceived by the client device 120 with be in different formats andsample rates, and the query fingerprint generator 125 creates, for someor all of the frames of the video content, a query fingerprint for eachframe that is scale independent and robust to different compressionartifacts, among other things. In some example embodiments, the queryfingerprint generator 125 may combine the query fingerprints of each ofthe frames to generate a query fingerprint of a block of frames (e.g.,multiple frames) of the video content.

The query fingerprint generator 125 may include a patch selection module210 configured and/or programmed to select multiple patches of the videocontent, such as patches associated with a displayed region of a frameor frames within the video content.

The query fingerprint generator 125 may also include a value calculationmodule 220 configured and/programmed to calculate a value for each ofthe selected multiple patches using an integral image technique, such asa technique that calculates the values using a summed area table orother data structure that generates a sum of values in a rectangulararea of a region.

For example, the patch selection module 210 may select patches, such asHaar-like features that are commonly used in object detection, ofregions of a frame or frames. The value calculation module 220 mayutilize the Haar-like features to generate and/or calculate a same valuefor objects in a frame, such as objects in a visual image of a frame),regardless of the relative size of the object. For example, the valuecalculation module 220 may generate values by approximating Gaussianfilters and their derivatives using a box filter (e.g., an average of aregion of the frame), wherein derivatives of the Gaussian filters arecreated by finding the differences in the box filters.

The query fingerprint generator 125, via the value calculation module220, may generate a query fingerprint by calculating the values ofHaar-like features, or patches, at different scales and in differentlocations of displayed regions of the frames of video content. FIG. 3Ais a display diagram 300 illustrating a first patch 310 applied to aframe 305 of video content. The patch 310 includes a middle region 317and outer regions 315. The value calculation module 220 may calculate avalue of the patch 310 by subtracting the middle region 317 from theouter regions 315, in order to determine what region of the patch 310has a higher amount of energy. FIG. 3B is a display diagram 320illustrating a second, different, patch 330 applied to a frame 305 ofvideo content. The patch 330 includes a middle region 337 and outerregions 335. The value calculation module 220 may calculate a value ofthe patch 330 by subtracting the middle region 337 from the outerregions 335, in order to determine what region of the patch 330 has ahigher amount of energy.

In some example embodiments, the patch selection module 210 selects 32patches, which include 8 patches for the first 8 regions of a frame and24 other patches of the left-right and top-bottom features of eachregion of the frame that results when the frame is divided into a 4×3grid of 12 regions. Of course, the patch selection module 210 may selecta different number of patches or a different configuration of patches.

In some example embodiments, the value calculation module 220 calculatesa query or reference fingerprint by utilizing integral image techniques.For example, the value calculation module 220 may calculate an integralimage, which is a summed image where each pixel is a cumulation ofvalues of the pixels above and to the left, as well as the currentpixel. The integral image technique may enable an efficient creation ofa fingerprint once an integral image is created, among other benefits.Further efficiencies may be realized by using Integrated PerformancePrimitive (IPP) libraries (created by Intel®), to create the integralimage, and/or by skipping columns and/or rows while generating theintegral image in order to process smaller amounts of data, among otherthings.

Using the integral images, the value calculation module 220 may thencalculate values for multiple regions or patches of a frame, such as byusing the following formula:

FP(i,j,w,h)=I _(int)(i,j)−I _(int)(i+w,j)−I _(int)(i,j+h)+I_(int)(i+w,j+h).

In this example, the value calculation module 220 calculates values forthe patches or regions of a frame as 16-bit integers, because the rangeis [−255, 255], which is outside the range of an 8-bit integer.Additionally, the calculated patches or regions of a frame may beweighted by their size, and the calculated values may be floating pointvalues. To capture this information, the values for the fingerprint arescaled to a full 16-bit range. However, in some example embodiments, thevalues may be quantized to 8-bits, reducing the storage space of agenerated fingerprint (e.g., from 64 bytes to 32 bytes), among otherthings.

In some example embodiments, the query fingerprint generator 125 mayverify whether one or more frames of video content includes pillar bars.Pillar bars are added to a frame of video content when the aspect ratioof a source device is different from an aspect ratio of a displaydevice. For example, in broadcasting, pillars are commonly added whenvideo content having 4:3 aspect ratio is displayed on a televisionhaving a 16:9 aspect ratio. In these scenarios, the query fingerprintgenerator 125 may verify whether frames include pillar bars, as includedpillar bars may affect the identification of fingerprints, among otherthings.

In some examples, in order to identify pillar bars in a frame of videocontent, the query fingerprint generator 125 may perform a binary searchof the calculated integral image to find edges of the image content. Forexample, the query fingerprint generator 125 may utilize a pillar baralgorithm in order to identify regions of actual video content within aframe, and not the pillar bars. The following is an example algorithmfor finding a left pillar bar in a frame of video content:

  function FindLeft(w, h)  x ← w/3  Δx ← w/3  while Δx > 1 do   Δx ←Δx/2   fp ← FP_(LvR)(0,0,x,h)   if |fp| < threshold then    x ← x + Δx  else    x ← x − Δx   end if  end while  return x end function

For every iteration, the algorithm may calculate the Haar-like feature(Left vs. Right) from the left edge of the image to the value of x, andwhen the area is generally flat (e.g., the same image content from Leftto Right), then x is moved further into the image; when there is asignificant difference, x is moved further out of the image. As thealgorithm runs, the amount that x is adjusted by (Δx) is halved, whichallows the algorithm to run in log(n) time to find the pillar box (e.g.,where n is ⅓ the width or height of the image).

FIG. 4 is a flow diagram illustrating an example method 400 forgenerating a query fingerprint, according to some example embodiments.The method 400 may be performed by the query fingerprint generator 125and/or the reference fingerprint generator 115 and, accordingly, isdescribed herein merely by way of reference thereto. It will beappreciated that the method 400 may be performed on any suitablehardware.

In operation 410, the query fingerprint generator 125 accesses a frameor block of frames of video content. For example, the query fingerprintgenerator 125 may access one or more frames of video content currentlyplaying on the client device 120.

In operation 420, the query fingerprint generator 125, or the referencefingerprint generator 115, selects multiple patches, of one or moreframes of video content, that are associated with a displayed region ofthe video content. For example, the patch selection module 210 selectsmultiple patches (e.g., 32 patches) of regions of a displayed frame, orpatches from multiple frames within a block of frames, among otherthings.

In operation 430, the query fingerprint generator 125 calculates a valuefor each of the selected multiple patches using an integral imagetechnique. For example, the value calculation module 220 calculatesvalues for each of the selected regions using the techniques describedherein. The calculated values may include values for large regions of aframe, small regions of a frame, regions that overlap (e.g., regionswhere a smaller patch overlaps a larger patch of a frame), and so on.

Thus, the query fingerprint generator 125 or the reference fingerprintgenerator 115 may generate a query fingerprint for a single frame and/ora block of frames of video content by selecting patches of regions (e.g.Haar-like features) of the frame or frames, calculating values for eachof the selected patches, and performing integral image techniques togenerate a query fingerprint using the calculated values for theselected patches, among other things.

Referring back to FIG. 2, the video identification system 150 includesthe index module 230, the fingerprint match module 240, and theidentification module 250, among other modules, which are configuredand/or programmed to match a query fingerprint to a known referencefingerprint.

In some example embodiments, the index module 230 is configured and/orprogrammed to query a database of known reference fingerprints of videocontent captured at a reference device, such as the watching station110. For example, the index module 230 may query an index stored withina database of the watching station 110, an index stored within the videoidentification system 150, and so on.

For example, the index module 230 may be configured to query an index ofquantized patch values of the known reference fingerprints. The indexmodule 230 may query an index of 8-, 16-, and/or 24-bit numbers that areassociated with single frames of the reference fingerprints. The indexmodule 230 may derive the numbers by quantizing one or more patch valuesof the reference fingerprints. For example, the index module 230 maynormalize fingerprint values by their absolute sum, by lognormalization, and so on.

The index module 230 may index some or all frames of the referencefingerprints using the best or better correlated values of large regionsof the frames. The index module 230 may create an index using highlycorrelated values of the full frame patches, because the featuresassociated with the best correlated values may represent the remainingpatches of a frame, among other things.

For example, when three regions of a frame are the best correlatedvalues, the index module 230 quantizes each of the three values to8-bits, such as by placing them in an evenly spaced histogram with amaximum and minimum limit, creating a 24-bit number. The index module230 may then utilize a reference fingerprint index of 24-bit numbers toquickly look up and/or identify frames, because there are only 81permutations per index to parse when attempting to match fingerprints.

In some example embodiments, the fingerprint match module 240 isconfigured and/or programmed to determine that a query fingerprintmatches at least one known reference fingerprint. For example, thefingerprint match module 240 may determine that a query fingerprintmatches at least one known reference fingerprint by determining that asimilarity between the query fingerprint and at least one of the knownreference fingerprints satisfies a predetermined threshold associatedwith a Tanimoto distance measurement, a Manhattan distance measurement,and/or other distance measurements associated with matching images orother visual-based content.

For example, the fingerprint match module 240 may compare a queryfingerprint to one or more reference fingerprints using the Tanimoto orthe Manhattan distance measurements, and determine that the fingerprintsmatch when the comparison indicates that the distance measurementsatisfies a predetermined threshold (e.g., is within a certain distancethreshold). Of course, the fingerprint match module 240 may utilizeother matching techniques in order to determine whether a queryfingerprint matches a reference fingerprint, such as Euclidean, Cosine,KL-Divergence and/or Itakura distance measurement techniques, amongother distance measurement techniques.

In some example embodiments, the video identification system 150 mayutilize various different block sizes (e.g., number of frames, or framerate) of fingerprints when matching a query fingerprint to a referencefingerprint. For example, the reference fingerprint may be set at 5frames per second (fps), and occupy approximately 560 KB/h of runtimegiven an 8-bit value (32 bytes/frame) for the fingerprint, and the queryfingerprint, which may include offset errors, may be set at 15 fps orhigher. In this example, a query of the index may involve queryingmultiple reference frames (e.g., 3 frames for a reference fingerprint)of the reference fingerprint index.

Thus, in some example embodiments, the video identification system 150may optimize a match rate for matching a query fingerprint to areference fingerprint, by modifying the block sizes of the fingerprints.For example, fingerprint match module 240 may match a query fingerprintto one second of reference video content, 0.5 second of reference videocontent, and so on.

In some example embodiments, the identification module 250 is configuredand/or programmed to identify video content captured at a client device120 based on a determination that a query fingerprint matches at leastone reference fingerprints. For example, the identification module 250may identify the name or title of the video content, a location withinthe video content currently being presented by the client device 120, achannel or broadcaster providing the video content, and so on.

FIG. 5 is a flow diagram illustrating an example method 500 foridentifying video content, according to some example embodiments. Themethod 500 may be performed by the video identification system 150 and,accordingly, is described herein merely by way of reference thereto. Itwill be appreciated that the method 500 may be performed on any suitablehardware.

In operation 510, the video identification system 150 accesses a queryfingerprint of video content that is based on multiple patches of aframe or frames of video content captured at a client device. Forexample, the video identification system 150 may access and/or receive aquery fingerprint from the query fingerprint generator 125 of the clientdevice 120.

In operation 520, the video identification system 150 queries a databaseof reference fingerprints associated with frames of known video content.For example, the index module 230 may query an index of quantized patchvalues of the known reference fingerprints.

In operation 530, the video identification system 150 determines thequery fingerprint matches at least one of the reference fingerprints.For example, the fingerprint match module 240 may determine that a queryfingerprint matches at least one known reference fingerprint bydetermining that a similarity between the query fingerprint and at leastone of the query reference fingerprints satisfies a predeterminedthreshold associated with a Tanimoto distance measurement, a Manhattandistance measurement, and/or other distance measurements associated withmatching images or other visual-based content.

In some example embodiments, the video identification system 150 maydetermine that two or more query fingerprints match two or morereference fingerprints, by using various distance measurementtechniques, among other things. For example, the video identificationsystem 150 may identify video content based on a match of multiplefingerprints.

In operation 540, the video identification system 150 identifies thevideo content captured at the client device based on the determinationthat the query fingerprint matches at least one of the referencefingerprints. For example, the identification module 250 may identifythe name or title of the video content, such as the name of a movie,television program, video clip, video game, and so on.

In some example embodiments, and in response to identifying the videocontent, the video identification system 150 may return an identifierfor supplemental content associated with the identified video content tothe client device 120. The client device 120 may use the receivedidentifier to access metadata, event-based content, and/or othersupplemental information or content that is associated with theidentified video content and/or associated with a time or location(e.g., a frame or block of frames) within the video content, that iscurrently being presented via the client device.

FIG. 6 is a block diagram illustrating components of a machine 600,according to some example embodiments, able to read instructions from amachine-readable medium (e.g., a machine-readable storage medium) andperform any one or more of the methodologies discussed herein.Specifically, FIG. 6 shows a diagrammatic representation of the machine600 in the example form of a computer system and within whichinstructions 624 (e.g., software) for causing the machine 600 to performany one or more of the methodologies discussed herein may be executed.In alternative embodiments, the machine 600 operates as a standalonedevice or may be connected (e.g., networked) to other machines. In anetworked deployment, the machine 600 may operate in the capacity of aserver machine or a client machine in a server-client networkenvironment, or as a peer machine in a peer-to-peer (or distributed)network environment. The machine 600 may be a server computer, a clientcomputer, a personal computer (PC), a tablet computer, a laptopcomputer, a netbook, an STB, a PDA, a cellular telephone, a smartphone,a web appliance, a network router, a network switch, a network bridge,or any machine capable of executing the instructions 624 (sequentiallyor otherwise) that specify actions to be taken by that machine. Further,while only a single machine is illustrated, the term “machine” shallalso be taken to include a collection of machines that individually orjointly execute the instructions 624 to perform any one or more of themethodologies discussed herein.

The machine 600 includes a processor 602 (e.g., a central processingunit (CPU), a graphics processing unit (GPU), a digital signal processor(DSP), an application specific integrated circuit (ASIC), aradio-frequency integrated circuit (RFIC), or any suitable combinationthereof), a main memory 604, and a static memory 606, which areconfigured to communicate with each other via a bus 608. The machine 600may further include a graphics display 610 (e.g., a plasma display panel(PDP), an LED display, an LCD, a projector, or a CRT). The machine 600may also include an alphanumeric input device 612 (e.g., a keyboard), acursor control device 614 (e.g., a mouse, a touchpad, a trackball, ajoystick, a motion sensor, or other pointing instrument), a storage unit616, a signal generation device 618 (e.g., a speaker), and a networkinterface device 620.

The storage unit 616 includes a machine-readable medium 622 on which isstored the instructions 624 (e.g., software) embodying any one or moreof the methodologies or functions described herein. The instructions 624may also reside, completely or at least partially, within the mainmemory 604, within the processor 602 (e.g., within the processor's cachememory), or both, during execution thereof by the machine 600.Accordingly, the main memory 604 and the processor 602 may be consideredas machine-readable media. The instructions 624 may be transmitted orreceived over a network 626 (e.g., network 190) via the networkinterface device 620.

As used herein, the term “memory” refers to a machine-readable mediumable to store data temporarily or permanently and may be taken toinclude, but not be limited to, random-access memory (RAM), read-onlymemory (ROM), buffer memory, flash memory, and cache memory. While themachine-readable medium 622 is shown in an example embodiment to be asingle medium, the term “machine-readable medium” should be taken toinclude a single medium or multiple media (e.g., a centralized ordistributed database, or associated caches and servers) able to storeinstructions (e.g., instructions 624). The term “machine-readablemedium” shall also be taken to include any medium that is capable ofstoring instructions (e.g., software) for execution by the machine(e.g., machine 600), such that the instructions, when executed by one ormore processors of the machine (e.g., processor 602), cause the machineto perform any one or more of the methodologies described herein. Theterm “machine-readable medium” shall accordingly be taken to include,but not be limited to, a data repository in the form of a solid-statememory, an optical medium, a magnetic medium, or any suitablecombination thereof.

Throughout this specification, plural instances may implementcomponents, operations, or structures described as a single instance.Although individual operations of one or more methods are illustratedand described as separate operations, one or more of the individualoperations may be performed concurrently, and nothing requires that theoperations be performed in the order illustrated. Structures andfunctionality presented as separate components in example configurationsmay be implemented as a combined structure or component. Similarly,structures and functionality presented as a single component may beimplemented as separate components. These and other variations,modifications, additions, and improvements fall within the scope of thesubject matter herein.

Certain embodiments are described herein as including logic or a numberof components, modules, or mechanisms. Modules may constitute eithersoftware modules (e.g., code embodied on a machine-readable medium or ina transmission signal) or hardware modules. A “hardware module” is atangible unit capable of performing certain operations and may beconfigured or arranged in a certain physical manner. In various exampleembodiments, one or more computer systems (e.g., a standalone computersystem, a client computer system, or a server computer system) or one ormore hardware modules of a computer system (e.g., a processor or a groupof processors) may be configured by software (e.g., an application orapplication portion) as a hardware module that operates to performcertain operations as described herein.

In some embodiments, a hardware module may be implemented mechanically,electronically, or any suitable combination thereof. For example, ahardware module may include dedicated circuitry or logic that ispermanently configured to perform certain operations. For example, ahardware module may be a special-purpose processor, such as a fieldprogrammable gate array (FPGA) or an ASIC. A hardware module may alsoinclude programmable logic or circuitry that is temporarily configuredby software to perform certain operations. For example, a hardwaremodule may include software encompassed within a general-purposeprocessor or other programmable processor. It will be appreciated thatthe decision to implement a hardware module mechanically, in dedicatedand permanently configured circuitry, or in temporarily configuredcircuitry (e.g., configured by software) may be driven by cost and timeconsiderations.

Accordingly, the term “hardware module” should be understood toencompass a tangible entity, be that an entity that is physicallyconstructed, permanently configured (e.g., hardwired), or temporarilyconfigured (e.g., programmed) to operate in a certain manner or toperform certain operations described herein. As used herein,“hardware-implemented module” refers to a hardware module. Consideringembodiments in which hardware modules are temporarily configured (e.g.,programmed), each of the hardware modules need not be configured orinstantiated at any one instance in time. For example, where thehardware modules comprise a general-purpose processor configured bysoftware to become a special-purpose processor, the general-purposeprocessor may be configured as respectively different hardware modulesat different times. Software may accordingly configure a processor, forexample, to constitute a particular hardware module at one instance oftime and to constitute a different hardware module at a differentinstance of time.

Hardware modules can provide information to, and receive informationfrom, other hardware modules. Accordingly, the described hardwaremodules may be regarded as being communicatively coupled. Where multiplehardware modules exist contemporaneously, communications may be achievedthrough signal transmission (e.g., over appropriate circuits and buses)between or among two or more of the hardware modules. In embodiments inwhich multiple hardware modules are configured or instantiated atdifferent times, communications between such hardware modules may beachieved, for example, through the storage and retrieval of informationin memory structures to which the multiple hardware modules have access.For example, one hardware module may perform an operation and store theoutput of that operation in a memory device to which it iscommunicatively coupled. A further hardware module may then, at a latertime, access the memory device to retrieve and process the storedoutput. Hardware modules may also initiate communications with input oroutput devices, and can operate on a resource (e.g., a collection ofinformation).

The various operations of example methods described herein may beperformed, at least partially, by one or more processors that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors may constitute processor-implemented modulesthat operate to perform one or more operations or functions describedherein. As used herein, “processor-implemented module” refers to ahardware module implemented using one or more processors.

Similarly, the methods described herein may be at least partiallyprocessor-implemented, a processor being an example of hardware. Forexample, at least some of the operations of a method may be performed byone or more processors or processor-implemented modules. Moreover, theone or more processors may also operate to support performance of therelevant operations in a “cloud computing” environment or as a “softwareas a service” (SaaS). For example, at least some of the operations maybe performed by a group of computers (as examples of machines includingprocessors), with these operations being accessible via a network (e.g.,the Internet) and via one or more appropriate interfaces (e.g., anapplication program interface (API)).

The performance of certain of the operations may be distributed amongthe one or more processors, not only residing within a single machine,but deployed across a number of machines. In some example embodiments,the one or more processors or processor-implemented modules may belocated in a single geographic location (e.g., within a homeenvironment, an office environment, or a server farm). In other exampleembodiments, the one or more processors or processor-implemented modulesmay be distributed across a number of geographic locations.

Some portions of this specification are presented in terms of algorithmsor symbolic representations of operations on data stored as bits orbinary digital signals within a machine memory (e.g., a computermemory). These algorithms or symbolic representations are examples oftechniques used by those of ordinary skill in the data processing artsto convey the substance of their work to others skilled in the art. Asused herein, an “algorithm” is a self-consistent sequence of operationsor similar processing leading to a desired result. In this context,algorithms and operations involve physical manipulation of physicalquantities. Typically, but not necessarily, such quantities may take theform of electrical, magnetic, or optical signals capable of beingstored, accessed, transferred, combined, compared, or otherwisemanipulated by a machine. It is convenient at times, principally forreasons of common usage, to refer to such signals using words such as“data,” “content,” “bits,” “values,” “elements,” “symbols,”“characters,” “terms,” “numbers,” “numerals,” or the like. These words,however, are merely convenient labels and are to be associated withappropriate physical quantities.

Unless specifically stated otherwise, discussions herein using wordssuch as “processing,” “computing,” “calculating,” “determining,”“presenting,” “displaying,” or the like may refer to actions orprocesses of a machine (e.g., a computer) that manipulates or transformsdata represented as physical (e.g., electronic, magnetic, or optical)quantities within one or more memories (e.g., volatile memory,non-volatile memory, or any suitable combination thereof), registers, orother machine components that receive, store, transmit, or displayinformation. Furthermore, unless specifically stated otherwise, theterms “a” or “an” are herein used, as is common in patent documents, toinclude one or more than one instance. Finally, as used herein, theconjunction “or” refers to a non-exclusive “or,” unless specificallystated otherwise.

What is claimed is:
 1. A method, comprising: accessing a queryfingerprint that is based on multiple patches of a frame of videocontent captured at a client device; querying a database of referencefingerprints associated with frames of known video content; determiningthe query fingerprint matches at least one of the referencefingerprints; and identifying the video content captured at the clientdevice based on the determination that the query fingerprint matches atleast one of the reference fingerprints.
 2. The method of claim 1,wherein accessing a query fingerprint that is based on multiple patchesof a frame of video content includes accessing a query fingerprint thatis based on a value calculated for the multiple patches of the frame ofvideo content using an integral image technique.
 3. The method of claim1, wherein accessing a query fingerprint that is based on multiplepatches of a frame of video content includes accessing a queryfingerprint that is based on a value calculated for multiple Haar-likefeatures of the frame of video content.
 4. The method of claim 1,wherein accessing a query fingerprint that is based on multiple patchesof a frame of video content includes accessing a query fingerprint thatis based on a value calculated for multiple Haar-like features of thevideo content, wherein each of the Haar-like features is associated witha unique location of a unique scale within the frame of video content.5. The method of claim 1, wherein querying a database of referencefingerprints includes querying an index of 24-bit numbers that areassociated with single frames of the reference fingerprints and derivedby quantizing patch values of the reference fingerprints.
 6. The methodof claim 1, wherein querying a database of reference fingerprintsincludes querying an index of quantized patch values calculated for thereference fingerprints.
 7. The method of claim 1, wherein determiningthe query fingerprint matches at least one of the reference fingerprintsincludes determining that a similarity between the query fingerprint andthe at least one of the reference fingerprints satisfies a predeterminedthreshold associated with a Tanimoto distance measurement.
 8. The methodof claim 1, wherein determining the query fingerprint matches at leastone of the reference fingerprints includes determining that a similaritybetween the query fingerprint and the at least one of the referencefingerprints satisfies a predetermined threshold associated with aManhattan distance measurement.
 9. The method of claim 1, furthercomprising: returning an identifier to the client device for the videocontent that is associated with metadata to be presented along with thevideo content via the client device.
 10. The method of claim 1, furthercomprising: returning an identifier to the client device for the videocontent that is associated with event-based information to be presentedalong with the video content via the client device.
 11. A system,comprising: a query fingerprint generator that includes: a patchselection module that is configured to select multiple patches of aframe of video content; and a value calculation module that isconfigured to calculate a value for each of the selected multiplepatches using an integral image technique; an index module that isconfigured to query a database of reference fingerprints associated withframes of known video content; a fingerprint match module that isconfigured to determine the query fingerprint matches at least one ofthe reference fingerprints; and an identification module that isconfigured to identify the video content captured at the client devicebased on the determination that the query fingerprint matches at leastone of the reference fingerprints.
 12. The system of claim 11, whereinthe selected multiple patches of the frame of video content are eachassociated with a displayed region of the frame of the video content.13. The system of claim 11, wherein the query fingerprint generator isconfigured to calculate values of multiple Haar-like features of theframe of video content.
 14. The system of claim 11, wherein the queryfingerprint generator is configured to calculate values of multipleHaar-like features of the frame of video content, wherein each of theHaar-like features is associated with a unique location and at a uniquescale of the frame of video content.
 15. The system of claim 11, whereinthe fingerprint match module is configured to determine that asimilarity between the query fingerprint and the at least one of thereference fingerprints satisfies a predetermined threshold associatedwith a distance measurement.
 16. A computer-readable storage mediumwhose contents, when executed by a computing system, cause the computingsystem to perform operations, comprising: accessing a query fingerprintthat is based on a value calculated for multiple patches of a frame ofvideo content using an integral image technique; querying a database ofreference fingerprints associated with frames of known video content;determining the query fingerprint matches at least one of the referencefingerprints; and identifying the video content captured at the clientdevice based on the determination that the query fingerprint matches atleast one of the reference fingerprints.
 17. The computer-readablestorage medium of claim 16, wherein accessing a query fingerprintincludes accessing a query fingerprint that is based on a valuecalculated for multiple Haar-like features of the video content.
 18. Thecomputer-readable storage medium of claim 16, wherein accessing a queryfingerprint includes accessing a query fingerprint that is based on avalue calculated for multiple Haar-like features of the video content,wherein each of the Haar-like features is associated with a uniquelocation of a unique scale within the frame of video content.
 19. Thecomputer-readable storage medium of claim 16, wherein querying adatabase of reference fingerprints associated with frames of known videocontent includes querying a database of reference fingerprints with twoor more accessed query fingerprints; and wherein determining the queryfingerprint matches at least one of the reference fingerprints includesdetermining the two or more accessed query fingerprints matches at leasttwo or more reference fingerprints.
 20. The computer-readable storagemedium of claim 16, wherein querying a database of referencefingerprints includes querying an index of quantized patch valuescalculated for the reference fingerprints.